Self-correcting Q-learning

The Q-learning algorithm is known to be affected by the maximization bias, i.e. systematic overestimation of action values, an important issue that has recently received renewed attention. Double been proposed as efficient mitigate this bias. However, comes at price underestimation in addition increased memory requirements and a slower convergence. In paper, we introduce new way address bias form "self-correcting algorithm" for approximating maximum expected value. Our method balances single estimator used conventional double Q-learning. Applying strategy results Self-correcting We show theoretically enjoys same convergence guarantees while being more accurate. Empirically, it performs better than domains with rewards high variance, even attains faster zero or low variance. These advantages transfer Deep Q Network implementation call DQN which outperforms regular on several tasks Atari 2600 domain.